{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card with a softly glowing rounded brain resting on top of it, a few small thought-bubble and gear props floating beside it to suggest reasoning, soft studio shadows, simple background, 16:9. */}

The first three posts in this series were about *making* things — the same RTX 3060
learning to generate [images](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/),
then [video](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/),
then [video with sound](/blog/2026/08/04-now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/).
This one is different. This time the card isn't drawing or filming anything — it's
*thinking*. I put a **27-billion-parameter language model on the same 12GB card**,
and it works.

The headline: **Qwen3.8-27B**, a reasoning model released only a few days ago, runs
locally on a consumer RTX 3060 that shipped back in **2021**. It answers questions, writes code,
reasons step by step — and it can even look at a picture and tell you what's in it.
The kind of model people assume needs a data centre, or at least a $2,000 graphics
card, runs on hardware a small business already owns.

## The machine (still the same one)

Same box as every post in this series: an RTX 3060 with **12GB of VRAM**, 64GB of
system RAM, Debian. Nothing upgraded. The card is the constant.

## Why this is surprising

A 27B model is *big*. The rule of thumb people quote is that you need a 24GB card
(an RTX 4090 or better) to run one at all — and its full-precision form is over
50GB, firmly data-centre territory. So "27B on 12GB" sounds like it shouldn't be
possible.

It is, because of two things working together: **quantization** and **offloading.**

## Fitting an 18GB model into 12GB

Quantization compresses the model's numbers to 4 bits each, shrinking it from 50GB+
down to about **18GB** with very little quality lost. That still doesn't fit in
12GB — but it doesn't have to. The card holds a bit more than half the model
(**about 57%**), and the rest lives in the 64GB of system RAM, streamed onto the
GPU as it's needed. The GPU does the heavy math; the spare layers wait in RAM.

The remarkable part is how *easy* it was. The video-with-sound model in the last
post took patches and careful memory tuning. This time it was genuinely two
commands — download, and run. The tool (Ollama) works out the GPU/RAM split
automatically.

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Model', 'Qwen3.8-27B (released this month)'],
            ['Size on disk', '~18GB (4-bit)'],
            ['Runs on GPU', '~57% on the card, ~43% streamed from RAM'],
            ['VRAM used', '11 of 12 GB'],
            ['Speed', '~4.7 words-per-second (reads about as fast as you do)'],
            ['Can it see images?', 'Yes — it is multimodal'],
            ['Best for', 'Questions, coding help, explanations, one-image analysis'],
        ]
    }
} />

Four-ish words a second isn't fast — you watch the answer type itself out, roughly
at reading speed. This is not the tool for churning through a thousand requests an
hour. It's the tool for asking one good question and getting a thoughtful answer,
privately, on your own machine, for the cost of the electricity.

## It reasons — out loud

Qwen3.8 is a *reasoning* model: before it answers, it thinks. Ask it a coding
question and you can watch it work the problem — considering approaches, catching
its own mistakes, checking its logic — then give a clean final answer. Ask it to
find the longest palindrome in a string, and it doesn't just spit out code; it
reasons about edge cases and complexity first. That visible thinking is a big part
of why these newer models are so much more useful than the ones from a year ago.

## And it sees

Here's the part that closes the loop on this whole series. The model is multimodal,
so I showed it one of the AI-generated animal photos from our image experiments and
asked: *is this a real photograph, or AI-generated?*

It nailed it. It described the scene correctly — a brown bear in a rushing river,
autumn forest blurred behind it — and then, unprompted, listed exactly *why* it
looked artificial: fur that was "suspiciously smooth and painted," the
"characteristic milky AI bokeh" in the background, "melted" rocks, a glassy eye
lacking real reflections. A local model, on a 12GB card, giving a genuinely sharp
critique of a locally-generated image. The tools in this series can now check each
other's work.

## Try it yourself

If you have an RTX 3060 (or any 12GB card) and [Ollama](https://ollama.com)
installed, this is the whole thing:

```bash
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M
```

The first run downloads the model; after that it's instant to start. Ask it
anything. To feed it an image, just put a file path in your message.

## Where this leaves a small business

Four posts, one unchanged computer. The same RTX 3060 went from a single 512px
still, to short video, to video with a soundtrack — and now to a reasoning model
that can answer questions, help write code, and read an image, all without a single
byte leaving the building.

That last point matters more than the speed. Running the model *locally* means your
questions, your customer data, your documents never touch someone else's server.
No per-message cloud bill, no data leaving your office, no dependency on a
subscription that can change its terms next quarter. For a lot of small businesses,
"good enough, private, and paid-for-once" beats "fastest" every time.

It isn't a data centre. It doesn't need to be.

*It's a five-year-old graphics card. What it can do keeps surprising me.*


0a2a03c8bcbf34fd700c45afe3e7e1d41f71654c0dd66b371de37159ffb3f7b5

Running Qwen3.8-27B, a brand-new 27B multimodal reasoning model, locally on a consumer RTX 3060 with only 12GB of VRAM: how an 18GB model fits, honest tokens-per-second, what it can and can't do, and the one command to try it yourself with Ollama.

The fourth in our RTX 3060 series. Qwen3.8-27B — a 27-billion-parameter reasoning model released this month — runs on the same 12GB card. Not in the cloud, not on a $2,000 GPU. One command, and it thinks, codes, and even sees.

Now It Thinks: A 27B Language Model on the Same RTX 3060

{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card as a tiny cinema, a small strip of film curling out of it, and a rounded speaker or sound-wave motif beside it emitting soft concentric rings to show the video now has audio; a little clapperboard prop, soft studio shadows, simple background, 16:9. */}

I ended the [last post](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/)
with a promise: the same RTX 3060 that learned to make short video would next try
to make video *with sound* — using **MiniMax H3**, an open-weights model that
generates its own audio. The weights landed publicly, I put them on the same
card, and this is what actually happened.

The headline first: it works. A single product photo goes in, and a few-second
clip comes out — the model turning, the fabric settling, a soft spoken line and
room tone — **audio the model generated itself**, not dubbed on afterwards. On a
12GB consumer card. That part genuinely surprised me.

## The machine (still the same one)

Same box as the [image retrospective](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
and the [video post](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/):
an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian. Nothing upgraded. The
card is the constant this whole series is built around.

## What "video with sound" means here

Most open video models are mute — you get moving pictures and add sound in an
editor. MiniMax H3 is different: video and audio come out of the *same* model,
generated together, so the soundtrack matches the scene. Describe waves and a
spoken line, and you get waves and a voice. That single capability is why it was
worth the effort.

## Fitting 54GB into 12GB

Here is the honest part. H3 is big: about **54GB of weights** — a 21GB video
model, a 27GB text encoder, and two decoders for picture and sound. That does
not fit in 12GB of VRAM. Not close.

It runs anyway because the weights live in the 64GB of system RAM and stream onto
the card a slice at a time, keeping just enough on the GPU to do the next step.
It is slow, but it finishes. Two smaller things also got in the way: the model is
new enough that its picture and sound decoders each had a small bug that had to
be patched before a clip would complete, and the card needed a healthy slice of
memory kept free so the compressed weights had room to unpack. Fiddly, but
one-time — once it was set up, it just ran.

## The result: a product clip that turns and talks

This is the workflow a small business would actually use: start from one product
photo, and let the model animate it — the person turns to camera, the knit
settles, a quiet line of audio plays — while the product stays readable. Here is
a real clip generated on this card. **Press play with the sound on** — the audio
came out of the model, not an editor.

<video controls playsInline preload="metadata" poster="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/sweater-poster.png" style={{ display: "block", width: "100%", maxWidth: "340px", margin: "1.5rem auto", borderRadius: "10px" }}>
  <source src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/sweater-clip.mp4" type="video/mp4" />
</video>

And the same clip laid out frame by frame — a natural turn from side-on to
camera, the knit texture and anatomy holding through the whole move:

![Frames from a MiniMax H3 clip: a model in a cream knit sweater turning to camera, generated with sound on an RTX 3060](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/filmstrip-sweater.png)

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Clip length', 'A few seconds (2-5s)'],
            ['Resolution', '512-640px'],
            ['Audio', 'Yes - native stereo, generated with the video'],
            ['Model size', '~54GB of weights, streamed from system RAM'],
            ['Time per clip', 'Roughly 20-27 minutes'],
            ['Best for', 'A short product clip with a voice or ambient soundtrack'],
        ]
    }
} />

Twenty-odd minutes for a few seconds is the real cost of streaming that much
weight through a small card. This is not something you batch out by the hundred
on a 3060 — it is something you run overnight for one good clip. But the clip has
sound, and it cost electricity.

## What I learned about the "AI look"

The most useful lesson wasn't about speed — it was about *why* AI video still
reads as AI, and how to fight it.

Image-to-video can never look better than the photo you start from; it inherits
that frame's every quirk and then adds motion on top. My first attempts started
from bright, high-contrast outdoor stills — and they had that unmistakable
"AI feel": waxy skin, hard outlines, the joins between body parts subtly wrong.
The motion clung to those hard edges and made them worse.

The clip above started from a **soft, studio-lit photo** instead — diffused
light, a plain background, gentle edges. And it held together: real fabric
texture, natural movement, believable anatomy through the whole turn. Same model,
same card, completely different result.

So the practical rule, if you try this yourself: **start from soft, evenly-lit
photos, not harsh high-contrast ones.** The model can't cling to hard edges that
were never there. It's the single biggest lever on how "real" the output looks —
bigger than any setting.

## Where this leaves a small business

Three posts, one unchanged computer. The same RTX 3060 went from struggling with
a single 512px [still](https://www.tyolab.com/prompts), to short
[silent video](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/),
to a short clip **with its own soundtrack** — a product turning to camera and
speaking a line, generated from one photo, locally, for the cost of the power it
drew.

It is not one-click, and it is not fast. But the gap between "I have a product
photo" and "I have a short product video with sound" has quietly closed on
hardware a small business can already afford. That's the whole point of this
series: the tools that used to need a studio now need a modest computer and a bit
of patience.

*The card hasn't changed in two years. What it can do has.*


b87386270e93d1cf860fa7cf05bfd9aba56d96224db423a19f429485654bc79d

Running MiniMax H3, an open-source video model with native audio, on a consumer RTX 3060 12GB: how 54GB of weights fit in 12GB, honest generation times, the quality limits, and turning one product photo into a clip with sound.

The third in our RTX 3060 series. MiniMax H3, an open-weights video model that generates its own audio, runs on the same 12GB card: a product clip with a real soundtrack, from a single photo, generated locally.

Now It Talks: Open-Source Video With Sound on the Same RTX 3060

{/* banner prompt: A playful but polished 3D clay-render, matte materials in sage green, cream and terracotta, of a stylized NVIDIA RTX 3060 graphics card acting as a small film projector on a simple background, casting a beam of light onto a tiny cinema screen that shows a strip of film frames; a couple of small clapperboard and film-reel props beside it, soft studio shadows, 16:9. */}

In the [last post](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
I traced how open-source *image* generation went from a 512px struggle to
photoreal Flux on one RTX 3060. I ended it with a tease about video. Here's the
payoff — and it's the part that genuinely surprised me.

Two years ago, generating video on a consumer graphics card wasn't really a
thing; the early attempts were a few flickery, warping frames. The **same RTX
3060** now produces short, coherent clips — and, more usefully for a small
business, it can take a **single product photo and turn it into a promo**.

## The machine (still the same one)

Same box as the [image post](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
and the [LLM benchmark](/blog/2026/05/11-64gb-ram-12gb-vram-the-honest-local-llm-benchmark/):
an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian 13. Nothing upgraded.

## What runs today

Two open models do the heavy lifting, both locally:

- **Wan 2.2** — higher quality, does text-to-video *and* image-to-video. Slower.
- **LTX-Video** — faster and lighter, good for quick drafts.

The 12GB card can't touch the giant cloud video models, but with the weights
quantised and the text encoder offloaded to system RAM, these run — a few
seconds of 480–832px video per clip.

## The trick that matters: one photo becomes a promo

This is the workflow a small business will actually use. Generate a product
photo (or use one you already have), then animate *it* — the model, the fabric,
the light all start moving, while the product stays put. Each strip below is
frames from a real clip, and each one started as a single still generated on
this same card.

A knit sweater: the model turns to camera, hair catching a breeze.

![Frames from the sweater promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-sweater.png)

Sunscreen — very on-brand for an Australian summer — the product held steady in
focus while the scene breathes:

![Frames from the sunscreen promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-sunscreen.png)

And eyewear, a slow confident turn in golden-hour light:

![Frames from the sunglasses promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-shades.png)

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Clip length', 'A few seconds (2-5s)'],
            ['Resolution', '480p to ~832px'],
            ['Time per clip (Wan 2.2)', 'Roughly 4-8 minutes'],
            ['Time per clip (LTX)', 'Under a minute for a short draft'],
            ['Best for', 'Image-to-video product promos, social clips'],
            ['Audio', 'Not yet — these models are silent'],
        ]
    }
} />

It's not one-click cinema. Big, fast motion still warps; the sweet spot is
*subtle* movement — a turn of the head, a breeze, a slow push — with the product
kept steady. You generate a couple of takes and keep the best, same as with
stills. But a usable, on-brand product clip in a few minutes, for the cost of
the electricity, on a card that couldn't manage one clean still two years ago —
that's the shift.

## What's next: sound

The current open video models are mute. The next one on our bench, **MiniMax
H3**, ships open weights *with* native audio and is reported to run on exactly
this class of hardware. When its weights land publicly, we'll put it on the same
3060 and see whether a small business can generate a product clip **with a
soundtrack** end to end — and we'll write up what actually happens.

For now, the takeaway from two posts and one unchanged GPU: the same modest
computer that makes studio-grade [stills](https://www.tyolab.com/prompts) now
makes short video too, and the gap between "I have a product photo" and "I have a
product video" has quietly closed.

*Next up: MiniMax H3 — open-weights video with sound — on the very same card.*


219f8b53de664082961a0b875c085a6d73339068a6cd5df6685f4c9fab0aea55

How open-source video generation runs on a consumer RTX 3060 12GB in 2026: Wan 2.2 and LTX-Video for text-to-video and image-to-video, real generation times, VRAM limits, and a worked example turning a product photo into a promo clip.

The follow-up to our image retrospective: the same RTX 3060 that struggled with a single 512px still two years ago now generates short video — including product promos animated from a single photo — with open models like Wan 2.2 and LTX-Video.

Now It Moves: Video Generation on the Same RTX 3060

{/* banner prompt: A playful but polished 3D render of a stylized RTX 3060 as a small factory on a simple background, using matte clay-like materials in a color story of sage green, cream, and terracotta; a conveyor belt on the left carries misshapen, glitchy figures, while a belt on the right carries perfectly formed photorealistic objects. */}

Eighteen months ago I wrote a tutorial on
[mastering human portraits with Stable Diffusion](/blog/2025/02-06-mastering-human-portraits-with-stable-diffusion-ai/).
Reading it back is a little painful. It opens with "trust me, this is hard," and
its list of search keywords reads like a bug report: *two heads, extra heads,
deformed face, extra eyes, disfigured, duplicate.* Those weren't edge cases —
they were what you fought on every render.

This week I ran open-source image generation on the **exact same computer**. Same
RTX 3060, same 12GB of VRAM. The only thing that changed is the software, and the
gap is hard to overstate.

## The machine (deliberately unchanged)

It's the same box from our [local-LLM
benchmark](/blog/2026/05/11-64gb-ram-12gb-vram-the-honest-local-llm-benchmark/):

- **CPU:** Intel Core i7-12700F
- **RAM:** 64GB DDR4
- **GPU:** NVIDIA GeForce RTX 3060 — 12GB VRAM
- **OS:** Debian 13

Holding the hardware constant is the whole point. Any improvement you see is the
open-source ecosystem getting better, not a new graphics card.

## Early 2025: Stable Diffusion 1.5, 512 pixels, and a war on anatomy

Back then, the state of the art you could actually run at home was Stable
Diffusion 1.5. My old parameters tell the story: 512×512, Euler a, CFG 7. To get
anything lifelike you leaned on community finetunes (RealisticVision, ChilloutMix,
F222), stacked LoRAs, and wrote paragraph-long negative prompts to *suppress* the
model's habit of adding a second head or a sixth finger. Then you generated a
dozen and kept the one that wasn't broken. It worked, but it was a craft — and a
grind.

## Mid 2026: the same card, a different league

Here's a portrait from the same 3060 today, straight out of the model —
no inpainting, no ControlNet, no clean-up:

![A photoreal portrait generated on the RTX 3060 in 2026](https://www.tyolab.com/api/tynion/images/f9f8fb60-d635-1048-cb10-c3492d7a4c4a/file)

Three things changed the game. **Flux** and modern **SDXL** finetunes replaced
SD 1.5 as the open baseline. They render natively at 1024px and up, and they get
hands, faces and proportions right the first time far more often than not.

They also learned to spell — the single clearest "we crossed a line" moment.
Readable text in an image was essentially impossible in 2025:

![A shopfront with legible hand-painted signage, generated locally](https://www.tyolab.com/api/tynion/images/3fa3d0ac-f3af-f6e7-952e-27b04f9b545e/file)

And the range on one card is wide — photoreal wildlife and clean anime styles
from the same machine, same afternoon:

![A photorealistic wildlife shot from the same GPU](https://www.tyolab.com/api/tynion/images/adbceadb-bc09-46f0-c8c7-bd76f312361f/file)

![A detailed anime illustration from the same GPU](https://www.tyolab.com/api/tynion/images/3d737b80-9cbb-186f-ce2c-c496fb918589/file)

## Then and now, side by side

<Table data={
    {
        headers: ['', 'Early 2025', 'Mid 2026'],
        rows: [
            ['Best model you can run', 'Stable Diffusion 1.5 + finetunes', 'Flux, SDXL finetunes'],
            ['Native resolution', '512 x 512', '1024px and up'],
            ['Anatomy', 'Fighting extra heads, hands, eyes', 'Right on the first try, mostly'],
            ['Text in the image', 'Gibberish', 'Often readable'],
            ['Keepers per ten tries', 'One or two', 'Most of them'],
            ['Effort per shot', 'Long negative prompts, LoRAs, inpainting', 'A plain-English sentence'],
            ['Video', 'Not on this card', 'A few seconds, locally'],
        ]
    }
} />

## The part that was pure science fiction in 2025: video

The same RTX 3060 now generates **video**. Wan 2.2 and LTX-Video produce short
text-to-video and image-to-video clips locally — a five-second scene in a few
minutes, on a card that two years ago struggled with a single 512px still. The
next open model on our list, MiniMax H3, even adds a synchronised soundtrack.

## What actually changed — and why it matters

It wasn't the GPU. Three shifts in the open-source world did the work:

- **Better base models.** Flux and SDXL are simply generations ahead of SD 1.5.
- **Quantisation.** Techniques like fp8 and 4-bit weights shrink 12-billion-parameter
  models enough to fit inside 12GB of VRAM with quality mostly intact.
- **Smarter memory management.** The tools now offload the heavy text encoders to
  system RAM, so the GPU only holds what it must.

Put together, a consumer card that launched years ago runs models that didn't
exist when you bought it — because the community learned to make them fit.

For a small business or a hobbyist, that's the real headline: the *same* modest
hardware budget now buys studio-grade stills and short video, from software
that's free and runs on your own desk. Every image in this post was generated on
that one 3060; you can browse and remix the prompts in our
[gallery](https://www.tyolab.com/prompts).

*Next up: putting MiniMax H3 — open-weights video with native audio — on the very
same card, and seeing how far 12GB can really go.*


cf0945211accc8aaf67c2caeafb240cb56b92b472152b7fc8a83a573245a77ac

A hands-on before-and-after: how open-source text-to-image models improved on the exact same RTX 3060 12GB between early 2025 and 2026 — Stable Diffusion 1.5 vs Flux and SDXL finetunes, resolution, anatomy, text rendering, hit-rate, and video.

We ran open-source image generation on the same RTX 3060 we used 18 months ago. The hardware is identical; the results are unrecognisable — from 512px portraits fighting extra heads to 1024px Flux, near-perfect anatomy, readable text, and even video.

Two Years of Open-Source Image AI on One RTX 3060


{/* banner prompt: A detailed architectural blueprint diagram of a system for local LLM inference, showing a stylized RTX 3060 GPU with its 12GB VRAM as a partitioned grid, connected by flow-lines to various model blocks (GGUF), with technical annotations and performance metrics in a clean, structured layout on a cyanotype blue background.

Additional instructions: with some construction workers watching */}

I've been building a local AI inference server at home. Not for any grand reason — mostly curiosity, a bit of privacy paranoia about sending everything to OpenAI, and the guilty pleasure of watching a model load onto my own GPU. After a few weeks of downloading models, writing benchmark scripts, and debugging scoring bugs at midnight, I have real numbers to share.

This is not a theoretical comparison. Every number here came from actual inference on my machine.

## The Hardware

- **CPU:** Intel Core i7-12700F (12 cores, 20 threads)
- **RAM:** 64GB DDR4
- **GPU:** NVIDIA GeForce RTX 3060 — 12GB VRAM
- **OS:** Debian 13
- **Inference engine:** [llama.cpp](https://github.com/ggerganov/llama.cpp) in router (multi-model) mode

The RTX 3060 with 12GB VRAM is the constraint that shapes everything. It's a solid consumer card — fast, quiet, widely available — but 12GB means you need to think carefully about which models you can load, at what quantization, and at what context length.

## The Setup: llama.cpp Router Mode

Rather than running a separate server process for each model, I use llama.cpp's `--models-dir` router mode. Point it at a directory of GGUF files, set `--models-max 2` (keep up to 2 models hot in VRAM at once), and it handles loading and unloading automatically per request. It exposes an OpenAI-compatible API at `http://localhost:8080/v1`.

The systemd service looks like this:

```ini
[Service]
ExecStart=/opt/llama.cpp/build/bin/llama-server \
    --host 0.0.0.0 --port 8080 \
    --models-dir /data/models \
    --models-max 2 \
    --threads 16 \
    --ctx-size 8192 \
    --n-gpu-layers 99 \
    --cont-batching \
    --flash-attn auto \
    --log-disable
```

`--n-gpu-layers 99` pushes every layer it can onto the GPU. `--flash-attn auto` enables Flash Attention when the model supports it. Models that don't fit entirely in VRAM will spill some layers to CPU — that's slow, and it shows up in the benchmark numbers.

## The Benchmark

I wrote a small Python script that tests each model on three task categories:

- **GSM8K** — 10 grade-school math word problems. Tests arithmetic reasoning.
- **MMLU** — 10 multiple-choice questions across medicine, history, philosophy, science, and CS. Tests general knowledge.
- **HumanEval** — 5 Python coding tasks (sum_to_n, palindrome check, fizzbuzz, count vowels, flatten). Tests code generation.

Each request uses `temperature=0` for determinism. Token speed (`t/s`) is measured from the `timings.predicted_per_second` field in the llama.cpp response.

### Handling Thinking Models

Several models — Qwen3, DeepSeek-R1-Distill, Qwen3.5 — are *thinking* or *reasoning* models that generate an internal chain of thought before giving a final answer. This created two real scoring headaches:

1. **llama.cpp routes Qwen3's thinking to a separate `reasoning_content` field** in the API response, leaving `content` empty. A naive scorer reading only `content` scores every Qwen3 response as 0%.

2. **256 max_tokens is nowhere near enough.** Thinking models use most of their budget on reasoning and never output a final answer. I bumped to 1024 tokens for GSM8K/MMLU and 2048 for coding tasks.

3. **MMLU first-match failure.** A reasoning model might mention option "A" in its thinking ("let's rule out A...") before concluding "the answer is B". Using `re.search` to find the first letter match gets the wrong answer. I switched to `re.findall` and take the *last* `[ABCD]` match.

The fixed scoring functions:

```python
def chat(model, prompt, max_tokens=512):
    r = requests.post(BASE_URL, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": 0,
    }, timeout=120)
    data = r.json()
    msg = data["choices"][0]["message"]
    # Qwen3 thinking models: answer is in reasoning_content, content is empty
    text = msg.get("content") or msg.get("reasoning_content", "")
    tps = data.get("timings", {}).get("predicted_per_second", 0)
    return text, tps

def score_mmlu(response, answer):
    stripped = strip_thinking(response)   # remove <think>...</think>
    for text in [stripped, response]:     # fallback to full text if strip leaves nothing
        matches = re.findall(r'\b([ABCD])\b', text.strip().upper())
        if matches and matches[-1] == answer:
            return True
    return False
```

Getting these details right took most of the iteration time. The first run showed Qwen3-8B at 0% on everything — not because it's a bad model, but because the scoring was broken.

## The Results

All models run as Q4_K_M quantization unless noted. 14B models were tested at `--ctx-size 4096` (reduced from 8192) to fit the KV cache alongside the ~8.4GB weights within 12GB VRAM.

```
Model                                    Params  GSM8K   MMLU   Code    Avg    t/s
─────────────────────────────────────────────────────────────────────────────────
Meta-Llama-3.1-8B-Instruct-Q4_K_M        8B     80%    80%   100%    87%   64.5
Qwen2.5-Coder-14B-Instruct-Q4_K_M       14B     80%    80%   100%    87%   35.8
Llama-3.2-3B-Instruct-Q4_K_M             3B     70%    80%   100%    83%  128.3
Qwen2.5-Coder-7B-Instruct-Q4_K_M         7B     40%   100%   100%    80%   69.3
Qwen_Qwen3-14B-Q4_K_M                   14B     50%    70%   100%    73%   33.4
Llama-3.2-1B-Instruct-Q8_0               1B     50%    50%   100%    67%  211.9
Qwen_Qwen3-8B-Q4_K_M                     8B     30%    70%   100%    67%   59.5
DeepSeek-R1-Distill-Qwen-7B-Q4_K_M       7B     80%    50%    20%    50%   66.2
Qwen3.5-9B-Q4_K_M                        9B     20%    70%    20%    37%   49.9
Qwen3.5-4B-Q4_K_M                        4B     30%    70%     0%    33%   77.1
MiMo-VL-7B-RL-q4_k_m                    7B     10%    50%    20%    27%   65.6
MiMo-7B-RL-Q4_K_M                        7B     10%    60%     0%    23%   64.2
THUDM_GLM-Z1-9B-0414-Q4_K_M              9B      0%     0%     0%     0%   51.3 †
─────────────────────────────────────────────────────────────────────────────────
DeepSeek-R1-Distill-Qwen-14B-Q4_K_M     14B    FAIL — wouldn't load at ctx=4096
openai_gpt-oss-20b-Q4_K_M               20B    FAIL — ~11.5GB, exceeds 12GB VRAM
Kimi-Linear-48B-A3B-BPW2.6              48B    FAIL — corrupt download
```

† GLM-Z1 uses non-standard thinking tags that our scorer doesn't strip. Its real capability is unknown from this run.

## What Surprised Me

### The 3B Llama Punches Hard

Llama-3.2-3B-Instruct at 128 t/s and 83% average is the sleeper hit. It's nearly as capable as the 8B model, runs at twice the speed, and fits comfortably in VRAM. If you're building something latency-sensitive — a coding assistant, a quick-reply bot — this is the model I'd reach for first.

### Qwen2.5-Coder-7B Is Surprisingly Well-Rounded

This model scored 100% on both MMLU *and* Code, and 40% on math. Named "Coder" but it's not just good at code — it's one of the most reliable general-purpose models in the lineup. At 69 t/s it's fast too.

### DeepSeek-R1-7B Is a Math Specialist, Not a Generalist

80% on GSM8K math — tied with the best models in the field. But 50% on MMLU and only 20% on coding. The reasoning focus pays off for math and pays a price everywhere else. Use it when you specifically need step-by-step numerical reasoning.

### 20B Models Don't Fit

The `openai_gpt-oss-20b` model is about 11.5GB at Q4_K_M quantization. That leaves only 500MB for the KV cache — not viable. It loaded partially with CPU offload, but responses were too slow and wrong. If you want a 20B+ model, you either need more VRAM or a heavily quantized version (Q2 or less), which carries its own quality tradeoffs.

### 14B Models Need Careful Context Sizing

At `--ctx-size 8192`, 14B Q4_K_M models fail to load — the weights (~8.4GB) plus the KV cache exceed 12GB. Dropping to `--ctx-size 4096` solved it for Qwen3-14B and Coder-14B. DeepSeek-R1-14B refused to cooperate even at 4096 — likely a memory fragmentation or architecture-specific issue.

### GLM-Z1 Needs a Different Scorer

GLM-Z1 almost certainly isn't as bad as our 0% score suggests. It uses thinking token formats that aren't `<think>...</think>` tags, so our strip function leaves garbage in the text. This is on the benchmark, not the model. Something to fix in a future run.

### Kimi-Linear-48B: So Close, Yet So Far

Kimi-Linear-48B is a Mixture-of-Experts model with 48B total parameters but only 3B active per token — meaning it should fit in 12GB VRAM and run at reasonable speed. At BPW2.6 quantization the file is about 8.1GB. We downloaded it twice. The first download (via `hf download`) produced a file with 6.9MB of zeros at the start — invalid GGUF magic. The second download via `wget` direct URL is in progress. Watch this space.

## The Hy-MT1.5 Translation Models

We also downloaded the full set of [Hy-MT1.5](https://huggingface.co/hy-mt) translation models from Tencent — 1.8B and 7B variants in multiple quantizations. These aren't benchmarked here because GSM8K/MMLU/HumanEval aren't the right tests for a translation model. A separate post will cover those with bilingual evaluation tasks.

## Practical Takeaways

If you're setting up a similar server, here's what I'd do differently from day one:

**1. Start with Llama-3.2-3B.** It's the best quality-per-token-per-second model in this lineup. Keep it hot in VRAM for low-latency tasks.

**2. Add Qwen2.5-Coder-7B for general use.** It's well-rounded across knowledge, code, and fast enough for interactive use.

**3. Use llama.cpp router mode from the start.** Running one server that hot-swaps models is far better than spinning up individual servers. You get an OpenAI-compatible API, automatic model loading, and a simple `/v1/models` endpoint to see what's available.

**4. Budget VRAM carefully for 14B models.** You need `--ctx-size 4096` (not 8192) to fit 14B Q4_K_M models. That's fine for most tasks, but be aware of the tradeoff.

**5. Validate GGUF downloads before benchmarking.** Check the first four bytes: they should be `47 47 55 46` (ASCII "GGUF"). A quick `od -A x -t x1z model.gguf | head -1` saves hours of debugging.

**6. Fix your thinking-model scorer before running anything.** The `reasoning_content` split in llama.cpp, the max_tokens floor, and the last-match MMLU scoring all need to be right or you'll think every thinking model is broken.

## What's Next

- Re-test GLM-Z1 with a proper thinking-tag stripper
- Kimi-Linear-48B, if the second download produces a valid file
- Translation quality tests for Hy-MT1.5
- A comparison post once larger VRAM cards (24GB+) are in reach

The benchmark script is straightforward Python using the OpenAI-compatible API — if you want to run it against your own setup, the core logic is about 100 lines. Happy to share it.

---

*Tested on Debian 13, llama.cpp built from source (May 2026), RTX 3060 12GB, i7-12700F, 64GB RAM.*


b669ccaaf96c4818fc77413de90012f54458d002144343d0ccad028881cec9dd

A hands-on benchmark of 13 local LLMs — Llama, Qwen3, DeepSeek-R1, Coder models and more — running on a home server with 64GB RAM and RTX 3060 12GB VRAM via llama.cpp. Includes GSM8K math, MMLU knowledge, and HumanEval coding scores, real token speeds, and an honest account of what doesn't fit.

We benchmarked 13 local LLMs on a machine with 64GB RAM and an RTX 3060 12GB using llama.cpp router mode — here's what scored well, what failed to load, and what surprised us.

Tag: Local Ai

Tag: Local Ai

Now It Thinks: A 27B Language Model on the Same RTX 3060

Now It Talks: Open-Source Video With Sound on the Same RTX 3060

Now It Moves: Video Generation on the Same RTX 3060

Two Years of Open-Source Image AI on One RTX 3060

13 Local LLMs, One RTX 3060: What Actually Runs (and What Doesn't)